HDR Video: Capturing and Displaying Dynamic Real-world Lighting

High Dynamic Range (HDR) video offers the possibility, for the first time, of capturing, storing, manipulating, and displaying dynamic real-world lighting. This gives a step change in viewing experience, for example the ability to clearly see the football when it is kicked from the shadow of the stadium into sunshine. An HDR video camera now exists which is capable of capturing 20 f-stops at full HD resolution (1920 × 1080) at 30 frames per second and commercial HDR displays are available. However, there are many significant challenges that still need to be overcome if HDR video is to be widely adopted and move from a niche research area into mainstream use. These include the need for high quality compression algorithms to cope with the enormous amount of data generated, the development of a common interface standard to facilitate widespread uptake, and even a definition of exactly what HDR is and what dynamic range might be considered “enough”. This paper investigates these challenges and highlights some of the key endeavours being undertaken to ensure HDR is the future of imaging technology. Introduction The human eye is capable of seeing detail in the wide range of colours and intensities that are presented to us in the lighting of the natural world, from moonlight to bright sunshine. Our eyes are constantly moving and adapting, and have a dynamic contrast ratio within any scene of nearly 1,000,000:1 (which equates to about 20 f-stops). Current imaging techniques are incapable of accurately capturing or displaying such a range of lighting. The remainder of the image outside the limited range, or Low Dynamic Range (LDR), of traditional cameras and displays, is either underor over-exposed. HDR imaging techniques, on the other hand, can cope with real-world lighting. This gives a step change in viewing experience, for example, the opportunity to clearly see the racing car as it enters and leaves the tunnel. In order for HDR to be widely adopted, it is fundamental that a complete HDR pipeline is available from capture, to manipulation if required, to delivery. This pipeline should strive to ensure that acquired HDR content is subsequently faithfully reproduced on an HDR display. HDR for still photography has been around for more than a decade. Such HDR techniques merge single exposure LDR images to create a picture that corresponds to our own vision, and thus meet our innate expectations. Where the dynamic range of the display matches that of the delivered lighting, the images can be displayed in a straightforward manner [1]. If this is not the case then the images need to be mapped to a lower dynamic range, known as tone-mapping, to ensure an enhanced viewing experience on a traditional display [2]. Video cameras capable of capturing higher dynamic ranges have been slowly emerging. From early research prototypes such as that of Unger et al [3], commercial cameras are now available which can acquire an increasing number of f-stops. For example, the Cannon 5D can capture 14 f-stops, while the recently announced Red Epic camera is claimed to obtain 18 f-stops. A more recent HDR video prototype capable of achieving 17 f-stops was proposed by Tocci et al [4]. The University of Warwick and the German high-precision camera company SpheronVR were instrumental in developing the world’s first HDR video camera, HDRv, capable of capturing 20 f-stops, full High Definition resolution (1920 × 1080 pixels) at 30 frames per second [5]. Commercial HDR displays are already on the market. After Dolby acquired the HDR display pioneers Brightside in February 2007, the technology was licensed to the Italian company SIM2. Their Solar 47, announced in 2009, has a resolution of 1920 × 1080 pixels, 2,206 LEDs in the back plane and a peak brightness of 2,000 cd/m. It is able to utilize full 16 bit processing and produce 65,536 shades per colour. Despite the increasing availability of HDR capture and display devices, complete pipelines are rare. One of the major remaining challenges is to cope with the large amounts of data that is generated by HDR video. For example, one minute of HDR video at 30fps full HD resolution requires 42GBytes of storage compared to four times less by traditional LDR video. Warwick HDR Pipeline The Warwick HDR system (wHDR) comprises a SpheronVR HDRv camera, and a number of HDR displays, including an array of 4 × BrightSide DR-37Ps, Figure 1. Figure 1: Warwick HDR video pipeline The A 3700 flat-panel Brightside DR37-P HDR displays have previously been characterized using a spectroradiometer to record spectral radiance, chromaticities and luminance [6]. The true increase in gamut of the display due to the additional LED layer was estimated. The colour space of the HDR displays have similar chromaticities to any other 8-bit per channel device, so there is no increased chromaticity resolution, but it is in the luminance dimension where the colour space is hugely increased. Table 1 shows the estimate of the number of new colours added to gamut for each different LED intensity and their sum. LED 1 32 64 128 192 255 Sum # 16,777,216 14,586,320 1,968,624 1,937,250 531,360 160,272 35,961,042 Table 1: The overall number of different colours for the HDR displays using 8bit+images, [6]. Multiple sensors in the HDRv camera use the same integration time to capture a scene simultaneously through one lens. As a consequence, the sensor needs different sensitivities to cover the whole dynamic range of the scene. The camera is capable of achieving 20 f-stops of dynamic range (1:1,000,000) and covering ~ 0.25 ... 250,000 cd/m. The key component though of wHDR is a compression algorithm which is capable of compressing the HDR video data by at least 100:1 with minimum perceptual difference. This algorithm has been filed as a patent and is the basis for a spinout company, goHDR Ltd. [7]. To date the compression algorithm has been tried out on a variety of HDR video footage, including RAW footage from the SpheronVR HDRv (20 f-stops), computer generated HDR animation (20 f-stops), OpenEXR footage from a Red One camera (12 f-stops), and a Cannon 5D (14 f-stops). Table 2 shows a summary of the results, while Figure 2 shows a frame in false colour from the HDR video footage of 567 frames (9,114MB), row 2 of Table 2. VDP>x% indicates the percent of pixels in the image which can be perceived differently with a probability greater than x% [8]. Source Frames RAW MB openEXR MB goHDR MB RAW: goHDR openEXR: goHDR VDP: P>95% 346 5,530 2,181 23 240.42 1 94.83:1 2.17% 567 9,114 3,840 27.6 330.20:1 139.13:1 1.68% 633 21,197 4,413 129 164.32:1 34.21 1 1.69% 751 13,619 4,648 27.2 500.71:1 170.88:1 4.74% HDRv 1,187 18,842 8,192 60.2 312.98:1 136.08:1 1.19% CGI 1,441 33,394 22,791 60 556.57:1 379.85:1 2.59% Red 789 N/A 4,618 50.8 N/A 90.91:1 5.01% 5D 987 N/A 3,220 53.5 N/A 60:1 3.68% Table 2: Compression ratios using goHDR algorithm Figure 2: False colour showing HDR in frame from HDRv Applications HDR video’s ability to capture and display dynamic realworld lighting provides a step change in viewing experience in a wide range of applications. In addition to the obvious benefits to the film and television industries, Figure 3, HDR video also enables other, previously unattainable, situations to be faithfully recorded and displayed. One of these applications is filming of surgical operations. The range of lighting present in surgery, include from the dark deep body cavities to the reflections of the bright operating theatre lights on the metal medical instruments, Figure 4. Figure 4: Lance Bell cartoon showing the “benefits” of filming medical operations in HDR. Figure 3: Tone mapped frame from the short film “Morgan Lovers”, which was shot and can be displayed entirely in HDR [9] Figure 5: Frame from HDR video of thorasic surgery. (left) exterior detail (right) detail inside the body with the diaphragm clearly visible. Figure 5 shows the detail both inside and outside the body that can be captured with HDR video. Furthermore, the wider range of colours available on an HDR display allows for a far more authentic viewing experience. The ability to capture the actual lighting of any given environment has given rise to rendering methods commonly labeled image-based lighting (IBL) techniques [10]. For using IBL, typically, a full sphere of lighting of a real scene is captured at a Figure 5: Tone mapped image of part of a scene after being captured with (top) 20 f-stops (bottom) 16 f-stops. Missing detail (and commercial opportunity) indicated in red. Figure 8: Enabling standards, the key piece of the HDR pipeline. Figure 6: Dynamic IBL with model of car relit by HDR video of explosion. The image shows 3 exposures from the footage plus false colour to indicate the dynamic range of the image. single point in an environment and stored as an HDR image, termed an environment map. This environment map is then used as the lighting inside a physically-based renderer in order to accurately light virtual objects with natural lighting. While most IBL implementations have used static environment maps, HDR video makes it possible to have dynamic natural lighting, enabling realistic animations for dynamic scenarios. Dynamic video also makes it more practical to capture static light fields which enables spatially varying rendering with natural lighting [11]. Figure 6 demonstrates an example of a prototype dynamic IBL showing a car lit by an explosion. 16 f-stops is not enough Despite the increasing ability to capture increasing amounts of the real world lighting, there are many in the film and television industry who believe that there is in fact no need to capture more than 16 f-stops; the dynamic range of traditional film. While it is true that many scenes do include less than 16 f-stops, there are many others for which 16-fstops is not enough. As Figure 7 shows, imposing an artificial limit on how many f-stops to consider may result in a loss of information which was present in the original scene, and in this picture, the loss of potential commercial revenue from a “product placement”. EU COST Action IC1005: HDRi There are still many barriers to be overcome if HDR video is to move from the research domain into wide commercial exploitation. Primary amongst these is the need to educate the market. EU COST Action IC1005, HDRi, started in May 2011 with the goals to coordinate HDR research across Europe, develop an international HDR community, play a key role in educating the market, and facilitate the wide spread adoption of HDR technology [12]. Funded by the European Science Foundation, the COST Action comprises leading HDR researchers, developers and users from 21 European countries. Together they will develop a new, efficient standard for HDR video across the entire HDR pipeline. Activities to support this goal include: short term scientific missions, summer schools, road-shows, a special user interest group for early adopters, and an annual international conference; the first of which is scheduled for October 2012. Conclusions HDR is on the cusp of becoming mainstream for most digital imaging applications. With its ability to acquire and display real world lighting more accurately and its enhanced colour capabilities, HDR video offers a step change in viewing experience. The availability of complete HDR pipelines from capture, through manipulation to display, such as Warwick’s system, offers a real potential to clearly identify and remove the remaining barriers currently restricting the commercial adoption of HDR technology. By pooling resources, through initiatives such as the EU Cost Action, “reinvention of the wheel” can be reduced and progress accelerated. Fundamental to this will be the adoption of an efficient standard specifically designed for HDR video. Carefully trialed through early adopters and introduced thoughtfully to minimise disruption of existing works flows, such a standard should play a key role in facilitating the commercial uptake and exploitation of HDR. Acknowledgements We would like to thank Glenn Mannion and his Medical Visualisation Team and Richard Steyn and the Thorasic Surgery Unit of Heartlands Hospital, Birmingham, UK for the collaborative endeavor to be the first to film a thorasic surgery in HDR. We would also like to thank Elmedin Selmanovic for the “16 f-stops is not enough” pictures.